Transmission control protocol (TCP) congestion control using transmission delay components

ABSTRACT

According to the present disclosure, methods and apparatus are provided to improve the Transmission Control Protocol (TCP) for data such as delay sensitive or bursty data. A maximum send window is adjusted using forward queuing delay and maximum bandwidth parameters. Reverse queuing delay and the number of packets drops are not factored into generation of the maximum send window. Network buffer occupation is bounded and a congestion window is effectively varied using rate shaping and gradual size increases based at least partially on the number acknowledged packets.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 12/840,206 filed Jul. 20, 2010, which itself is a continuationof U.S. patent application Ser. No. 11/291,201, now U.S. Pat. No.7,760,633 issued May 31, 2007, both of which are incorporated herein byreference.

BACKGROUND

1. Field

The present disclosure generally relates to the congestion control. Morespecifically, the present disclosure provides techniques and mechanismsfor improving the transmission control protocol (TCP), particularly fortransmitting data such as storage application data.

2. Description of Related Art

TCP provides reliability, network adaptability, congestion control andflow control. Reliability is generally provided by using mechanisms suchas sequence numbers to enable retransmission. Network adaptability andflow control are generally provided by using mechanisms such as windows.A window limits the amount of data that can be transmitted onto anetwork.

Conventional TCP congestion control mechanisms work well for many typesof data transmissions. However, conventional TCP congestion controlmechanisms often do not work adequately for delay sensitive or burstydata, such as data associated with an Internet Protocol (IP) StorageApplication, especially when the bandwidth, delay, and optimal windowsizes of a connection are large. In one example, TCP does not workadequately for transferring data associated with Storage Area Networks(SANs). Some improvements to TCP such as FastTCP, described in “FastTCP:Motivation, Architecture, Algorithms, Performance” by Chen Jin, DavidWei, and Steven Low, IEEE Infocom, March 2004, Hong Kong, address someconcerns associated with TCP but still have a number of limitations.

Consequently, it is desirable to provide techniques for improving TCP toallow more effective and efficient transmission of data such as delaysensitive and bursty data, in order to greatly reduce the probability ofpacket drops while minimizing delay.

SUMMARY

According to the present disclosure, methods and apparatus are providedto improve the Transmission Control Protocol (TCP) for data such asdelay sensitive or bursty data. A maximum send window is adjusted usingforward queuing delay and maximum bandwidth parameters. Reverse queuingdelay and the number of packets drops are not typically factored intogeneration of the maximum send window, even though recent packet dropsdo cause the send window to decrease. By controlling the maximum sendwindow size using an estimate of the forward congestion delay, networkbuffer occupation is bounded and a congestion window is effectivelyvaried using rate shaping. The congestion window size graduallyincreases based at least partially on the number of recentlyacknowledged bytes.

In one embodiment, a technique for performing congestion control using atransmission control protocol (TCP) is provided. A forward delaycomponent of a round trip time associated with sending data associatedwith a flow from a source node and receiving an acknowledgment from adestination node is determined using the transmission control protocol(TCP). A maximum window is adjusted by using the forward delay componentand an observed minimum round trip time.

In another embodiment, a network device is configured to performcongestion control using a transmission control protocol (TCP). Thenetwork device includes an interface and a processor. The interface iscoupled to an Internet Protocol (IP) network. The processor is operableto determine a forward delay component of a round trip time associatedwith sending data associated with a flow and receiving an acknowledgmentfrom a destination node. The destination node is connected to theinterface using the transmission control protocol (TCP). The processoris also configured to adjust a maximum window by using the forward delaycomponent and an observed minimum round trip time.

In some embodiments, when the estimated forward congestion delayincreases, the window size is decreased even before any packet drop isdetected.

A further understanding of the nature and advantages of the presentdisclosure may be realized by reference to the remaining portions of thespecification and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, whichare illustrative of specific embodiments of the present disclosure.

FIG. 1 is a diagrammatic representation showing network nodes that canuse the techniques of the present disclosure.

FIG. 2 is a diagrammatic representation showing a TCP transmissionstream.

FIG. 3 is a diagrammatic representation showing a TCP sliding window.

FIG. 4 is a flow process diagram showing one technique for updating awindow.

FIG. 5 is a diagrammatic representation of a device that can use thetechniques of the present disclosure.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Reference will now be made in detail to some specific embodiments of thedisclosure including the best modes contemplated by the inventors forcarrying out the disclosure. Examples of these specific embodiments areillustrated in the accompanying drawings. While the disclosure isdescribed in conjunction with these specific embodiments, it will beunderstood that it is not intended to limit the disclosure to thedescribed embodiments. On the contrary, it is intended to coveralternatives, modifications and equivalents as may be included withinthe spirit and scope of the disclosure as defined by the appendedclaims.

For example, the techniques of the present disclosure will be describedin the context of the transmission control protocol (TCP). However, itshould be noted that the techniques of the present disclosure can beapplied to different variations and flavors of TCP as well as toalternatives to TCP and other network protocols that have a congestioncontrol component. In the following description, numerous specificdetails are set forth in order to provide a thorough understanding ofthe present disclosure. The present disclosure may be practiced withoutsome or all of these specific details. In other instances, well knownprocess operations have not been described in detail in order not tounnecessarily obscure the present disclosure.

Furthermore, techniques and mechanisms of the present disclosure willsometimes be described in singular form for clarity. However, it shouldbe noted that some embodiments can include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. For example, a processor is used in a variety of contexts.However, it will be appreciated that multiple processors can also beused while remaining within the scope of the present disclosure.

The transmission control protocol (TCP) is a transport layer protocolthat provides full-duplex, stream-oriented, connections allowing forreliable transmissions, network adaptation and flow control. TCPprovides transmission of streams of bytes in full-duplex. Traffic flowsin both the forward and reverse directions. Only during connection startand close sequences can TCP exhibit asymmetric behavior. Datatransmissions are organized into different connections.

Reliability is provided using sequence numbers to track what data hasbeen transmitted and received for each particular connection. TCParranges for retransmission if it determines that data has been lost.Plain TCP learns about delay characteristics associated with a networkand attempts to adjust its operation to maximize throughput by adjustingits retransmission timer. TCP uses 32-bit sequence numbers that identifybytes in the data stream. Each TCP packet includes the starting sequencenumber of the data in that packet, and the sequence number (alsoreferred to as an acknowledgment number) of the last byte received fromthe remote peer. Forward and reverse sequence numbers are independent,and each TCP peer tracks both its own sequence numbering and thenumbering being used by the remote peer. TCP also uses a number of flagsto manage connections.

TCP provides adaptability and flow control by using windows. To avoidcontinuously overflowing various network buffers, TCP attempts to managethe amount of data transmitted onto a network. In typical instances, awindow limits the amount of data not yet acknowledged that can betransmitted onto a network. When the window is full of data transmittedby not yet acknowledged, no other data can be sent. When anacknowledgment is received, the window slides and additional data can besent. If no acknowledgment is received after a predetermined time outperiod, the oldest packet is assumed to have been lost and the data isretransmitted. Some TCP flavors use more sophisticated techniques thattake advantage of the selective acknowledgement options.

In some instances, TCP varies the size of the window based on whether ornot an acknowledgment is received. Any window that is varied in sizebased on transmission characteristics is referred to herein as acongestion window. In one example, a congestion window grows by onesegment every time a positive acknowledgment is received. Consequently,the sender not only can send new data based on the acknowledgment beingreceived but can also send new data based on the increased window size.However, the scheme is often too aggressive, as the quickly growingwindow size will eventually cause too much data to be transmitted ontothe network and lead to packet drops. Similarly, the congestion windowtypically shrinks to a single segment every time the sender is idle formore than a retransmission timeout.

The congestion window then gradually grows based on successfultransmissions. The congestion window grows linearly if the TCPCongestion Avoidance scheme is being used and the congestion windowgrows exponentially if the TCP Slow Start scheme is being used.Congestion Avoidance and Slow Start are described in RFC 2001. However,the growth of the congestion window is often slow, particularly becausethe congestion window shrinks to a very small size. Consequently,neither Congestion Avoidance nor Slow Start is effective for bursty datasuch as data from storage applications in storage area networks. Theconservative growth of the congestion window may also not be suitablefor delay sensitive data such as real-time video data.

Furthermore, TCP typically performs no traffic shaping of any sort. As acongestion window is growing in size, bursty traffic can be transmittedsuddenly onto a network without regard to any traffic shapingparameters. Consequently, bursty traffic can end up flooding networkqueues and at the minimum creating unnecessary delay affecting othertraffic and risking buffer overflow.

FastTCP, developed by the Networking Group led by Steven Low at CalTech,is described in “FastTCP: Motivation, Architecture, Algorithms,Performance” by Chen Jin, David Wei, and Steven Low, IEEE Infocom, March2004, Hong Kong. FastTCP updates the maximum transmit window size usinground trip times and the current value of the window. Window sizes fortransmissions from a source for a particular flow are adjusted usinground trip time. Packets drops are no longer considered as a primaryfactor in adjusting window sizes, even tough FastTCP honors the standardTCP semantics and shrinks window sizes when packet drops are detected.Round trip times provide a workable mechanism for adjusting window sizesand transmit rates.

However, the techniques of the present disclosure recognize that roundtrip times are sensitive to congestion happening in the forwarddirection from a source to a destination as well as congestion happeningin the reverse direction from the destination back to the source. It isrecognized that it would be preferable to consider only forwarddirection congestion, since forward direction congestion or forwardqueuing is what should affect window sizes and transmission rates from asource. Conventional TCP and FastTCP both are sensitive to reversedirection congestion. Reverse direction congestion or reverse queuingshould only affect window sizes and transmission rates from othernetwork nodes, including the peer.

FastTCP also does not put any bound on network buffer consumption. Infact, aggregate network buffer usage grows linearly with the number ofFastTCP connections in a network. The techniques of the presentdisclosure recognize that this is not desirable because it provides thatthere is no upper bound to the amount of buffer required in networknodes. In the presence of a single bottleneck, all FastTCP connectionscongest the same queue, and if the queue length is insufficient toaccommodate them, drops still occur, defeating some of the primarybenefits of FastTCP.

Consequently, the techniques of the present disclosure adjust windowsizes using at least the forward component of network delay. That is,forward direction delay associated with queuing for transmission from asource to a destination is considered a primary factor in adjustingwindow sizes. A maximum window size, as well as a congestion windowsize, is calculated using values associated with forward directiondelay. The amount of buffer consumed in a network by a particular flowis controlled in order to limit the total amount of buffer usage.Transmission rates are also controlled using traffic shaping and managedwindow size changes. A variable rate shaper is used to pace packetintroduction onto the network to avoid bursts, such as bursts associatedwith data from a disk array. The variable rate shaper maximum rate iscalculated based on the maximum window size and the measured round triptime. The congestion window is controlled based on the forward componentof the network delay, as opposed to the full round trip time. The amountof bottleneck buffer consumed by each connection is also controlled inorder to limit the total amount of buffer usage. Buffer usage no longergrows linearly based on the number of flows.

The techniques and mechanisms of the present disclosure provide a numberof improvements to conventional TCP and FastTCP. Various embodiments ofthe present will be referred to herein as Internet Protocol Storage(IPS) FastTCP. The techniques and mechanisms of the present disclosureprovide relatively smooth traffic flow, reduced probability of packetdrops and required retransmissions, reduced bottleneck queuing, and moreoptimal bandwidth usage. Transmission windows are not distorted bycongestion on the reverse path and total buffer usage in the network isbounded. The techniques and mechanisms of the present disclosure can beapplied to a variety of networks, including IP storage associatednetworks, where packet drops are particularly undesirable. Thetechniques of the present disclosure increase overall performance byreducing the probability of drops and increasing overall bandwidthusage.

FIG. 1 is a diagrammatic representation showing a network topology thatcan use the techniques of the present disclosure. A storage area network101 includes hosts 121 and 123 along with storage node 125. Storage node125 may include a disk or tape array. The storage area network 101 canalso include multiple fibre channel switches. The storage area network101 is coupled to an IP network 103 through a tunneling switch 111.Storage area network 105 includes host 127, storage 129, as well asother fibre channel switches and tunneling switch 113. The tunnelingswitches 111 and 113 allow the formation of a tunnel to transmit storagenetwork data over an IP network 103.

According to various embodiments, improvements to TCP can be implementedat any source originating traffic or at any destination receivingtraffic. For example, improvements to TCP can be implemented at hosts ina storage area network. In another example, improvements to TCP can beimplemented at tunneling switches connecting storage area networks to anIP network. The techniques can be implemented anywhere TCP isimplemented. TCP typically allows the transmission of data usingwindows.

FIG. 2 is a diagrammatic representation showing a window. A data streamis separated into different parts. Portion 211 is data that has beensent and acknowledged. Portion 213 is data has been sent but not yetacknowledged. In some examples, this portion 213 includes data that hasbeen retransmitted one or more times. This part is often referred to asthe flight size 203. Another part of the data stream is referred to asthe usable window 205. The usable window is the portion 215 that can besent but has not yet been sent over the network.

In some examples, the usable window has a non-zero size when space isavailable in window 201. When the usable window 205 diminishes to nearly0 and the flight size 203 encompasses nearly all of window 201, noadditional data can be sent. The flight size 203 and the usable window205 together are referred to as a window 201, transmission window, orcongestion window. The data stream can also include data 217 that cannot yet be sent over the network.

FIG. 3 is a diagrammatic representation showing a sliding window. Asdata is acknowledged, a previous window 303 slides over to a currentwindow 301 position. As data is transmitted and acknowledged, thecurrent window continues to move to incorporate more data that has notbeen sent. For example, an acknowledgment may be sent for every 2packets received by a destination. Each acknowledgment detected canshift the current window 301 over a predetermined amount. The windowtypically slides by the amount of data that has been acknowledged. Thewindow typically changes size based on slow start and congestion controlmechanisms.

FIG. 4 is a simplified flow process diagram showing a technique foradjusting a congestion window. It should be noted that the flowprocesses depicted herein are merely examples of techniques of thepresent disclosure. A variety of other techniques can also be applied.Some of the processes depicted are optional while other processes may beadded while remaining within the scope of the present disclosure. Insome instances, details associated with the process operations may notbe described in order not to unnecessarily obscure the presentdisclosure.

At 401, it is determined if space is available in a window. According tovarious embodiments, space is available if a window is not full of datasent but not yet acknowledged. If no space is available, the senderwaits for acknowledgments corresponding to sent packets or for sentpacket time outs. If space is available in a window, data is transmitted405. In one example, data is continuously transmitted as long as spaceis available in a window. Various counters, timers and sequence numberscan be maintained to allow for reliable retransmission. As data istransmitted, a decreasing amount of space is available in the window fortransmission until acknowledgments are received.

If an acknowledgment is received at 411, the window now has more spaceto incorporate new data to be transmitted. In one example, if anacknowledgment is received at 411, the window slides to include new datafor transmission at 417. In some instances, the window is enlarged byone segment. However, if no acknowledgment is received after one or moreretransmit attempts at 413, the window is collapsed to a very smallsize, often to one or two segments at 415. This is often a very drasticremedy for addressing dropped packets. This is an effective solution forconventional protocols such as the file transfer protocol (FTP), whichis more concerned focused towards effectively transferring a file thanit is with real-time handling of bursty data.

However, the techniques of the present disclosure recognize that thewindow need not be adjusted as drastically. The techniques of thepresent disclosure provide mechanisms for intelligently adjusting thewindow size, based on bandwidth availability and forward delaycomponents. According to various embodiments, a congestion window isadjusted every time TCP input is invoked. A maximum send window(snd_hiwat) is adjusted once per round trip time, after the previouswindow has been fully acknowledged.

FastTCP adjusts window sizes based on the delay associated with networkbuffering and changes the window size to the smaller of either twice thecurrent window size or a window size modified by a control functiondependent on round trip times. FastTCP adjusts window sizes using thefollowing equation:

$\begin{matrix}{{{win}(k)} = {\min\left\{ {{2{w\left( {k - 1} \right)}},{{{\left( {1 - \gamma} \right){w\left( {k - 1} \right)}} + {{\gamma\left( {{{w\left( {k - 1} \right)}\frac{baseRTT}{R\; T\; T}} + {\alpha\left( {w,{delay}} \right)}} \right\}}\mspace{79mu} R\; T\; T}} = {{{baseRTT} + {{delay}\mspace{79mu} 0}} < \gamma \leq {1\mspace{79mu}\left\{ {\begin{matrix}\left. {{delay} > 0}\Rightarrow{{\alpha\left( {w,{delay}} \right)}\overset{\sim}{a}} \right. \\\left. {{delay} > 0}\Rightarrow{{\alpha\left( {w,{delay}} \right)}\alpha\; w} \right.\end{matrix},} \right.}}}} \right.}} & (1)\end{matrix}$where w is a congestion window; RTT is round trip time; baseRTT is theobserved minimum round trip time; delay(t) is a delay associated withnetwork queuing; and y is an empirically determined control constraintused to govern congestion window changes.

Once some extra delay is detected, that is, when the average RTT becomeslarger than the baseRTT, the parameter α is held constant. It can beshown that on equilibrium α is the average sum of the buffer occupationfor the connection. Since the metric used to determine congestion is theround trip time, FastTCP is sensitive to congestion happening in bothdirections of the path between two network endpoints. The TCP flowsconverge to a global stability point, but the configuration of flowsdoes not necessarily maximize usage of the available bandwidth. Forexample, the resulting delay may all be associated with delay on thereverse path or the return path. The techniques of the presentdisclosure recognize that delay on the return path or the reverse pathshould not affect congestion window sizes.

In the presence of N FastTCP connections all using the same α, the totalaverage buffer occupation is up to N*α. This is not desirable because itessentially implies that there is no upper bound to the amount of bufferrequired in network nodes, unless the max number of concurrentconnections is known in advance. Even when an upper bound on the numberof connections is known, the required buffer occupation might be biggerthan the affordable buffer size. In the presence of common singlebottleneck congestion, all the connections congest the same queue, andif the maximum queue length is insufficient to accommodate them, dropshappen, thereby defeating the usefulness of FastTCP.

The forward delay component is associated with a reverse delay componentand in many instances, it will be recognized that simple substitutionswill allow calculation of a maximum window using forward delaycomponents. Reverse delay can be calculated in a variety of manners. Inone example, accurately synchronized source and destination nodes willallow determination of forward and reverse delays. In other examples,reverse delay can be determined using the techniques and mechanismsdescribed in U.S. patent application Ser. No. 11/291,251, titled,“Methods and Apparatus for Determining Reverse Path Delay,” by GuglielmoM. Morandin, now U.S. Pat. No. 7,643,430 issued Jan. 5, 2010, theentirety of which is incorporated by reference for all purposes.

According to various embodiments, reverse delay is calculated bymaintaining accurate measurements at a source node and receivingtimestamp packets from a destination node. Any packet or acknowledgeincluding a timestamp is referred to herein as a timestamp packet. Insome instances, a timestamp packet is a typical acknowledgment withtimestamp information. The destination node has a timestamp speed atwhich timestamp packets are incremented, for example 10 ms or 1 ms. Whena timestamp packet is received, it is determined whether the timestampexpected in the packet is different from the timestamp value included inthe packet. In one particular example, the reverse delay is thetimestamp value subtracted from the timestamp expected all divided bythe timestamp speed.

The techniques and mechanisms of the present disclosure provide amaximum window that is used to adjust a congestion window. It should benoted that the maximum window is merely a target window and notnecessarily an absolute maximum. According to various embodiments, themaximum window is adjusted using the forward delay component associatedwith a round trip time. Other factors such as an observed minimum roundtrip time or a maximum bandwidth can also be considered.

According to various embodiments, the maximum window is calculated usingthe following equation:

$\begin{matrix}{{{{snd\_ hiwat}(t)} = {{{{cwnd}\left( {t - 1} \right)}\frac{{baseRTT} + {{rdelay}\left( {t - 1} \right)}}{R\; T\;{T\left( {t - 1} \right)}}} + {\alpha\left( {{max\_ bw},{{qdelay}\left( {t - 1} \right)}} \right)}}}{{R\; T\;{T(t)}} = {{baseRTT} + {{qdelay}(t)} + {{rdelay}(t)}}}} & (2) \\\left\{ \begin{matrix}\begin{matrix}{\left. {{L - {{max\_ bw}*{qdelay}}} > 0}\Rightarrow\alpha \right. = {\left( {L - {{max\_ bw}*{qdelay}}} \right) \cdot K_{p}}} & {0 < K_{p} \leq 1}\end{matrix} \\{\left. {{L - {{max\_ bw}*{qdelay}}} \leq 0}\Rightarrow\alpha \right. = 0}\end{matrix} \right. & (3)\end{matrix}$where cwnd is a congestion window; snd_hiwat is the maximum window;RTT(t) is the round trip time, baseRTT is the observed minimum roundtrip time; rdelay(t) is a reverse delay component of the round triptime; qdelay(t) is the forward delay component of the round trip time; Lis a target buffer occupation; max_bw is a maximum bandwidth; and K_(p)is an empirically determined proportional controller constant between 0and 1.

Equation 2 can also be rewritten in the following form:

$\begin{matrix}{{{{snd\_ hiwat}(t)} = {{{{cwnd}\left( {t - 1} \right)}\frac{{bRTT}\left( {t - 1} \right)}{{{bRTT}\left( {t - 1} \right)} + {{qdelay}\left( {t - 1} \right)}}} + {\alpha\left( {{max\_ bw},{{qdelay}\left( {t - 1} \right)}} \right)}}}\mspace{79mu}{{{bRTT}(t)} = {{baseRTT} + {{rdelay}(t)}}}} & (4)\end{matrix}$

In some examples, the maximum window is evaluated every time a currentwindow is acknowledged. The baseRTT is used to determine the networkdelay associated with a network free of traffic. The techniques andmechanisms of the present disclosure recognize that congestion delay ismerely one source of delay in the network. Other sources of delayinclude router processing and forwarding delays, source and destinationdelays, latency delays, etc. The techniques of the present disclosureprovide mechanisms for measuring forward delay independent of baseRTT.

With the introduction of rdelay in the control equation, the techniquesand mechanisms of the present disclosure are immune to congestionoccurring in the reverse direction. The equilibrium point is achievedwhen each connection maintains a bytes queued in the forward direction.

The max_bw or maximum bandwidth is a static property of the variousnetwork nodes and link in a network and does not depend on the amount oftraffic present in the network. Consequently, the max_bw can beempirically determined and/or manually configured. The value can also beestimated and autoconfigured in other examples.

The FastTCP buffer utilization is proportional to the number of activeflows. However, the techniques of the present disclosure can impose anupper bound on buffer usage by using a variable α function. This avoidsa significant drawback of FastTCP. The function a determines theequilibrium point for optimized distribution of network bandwidth. Ingeneral, it is desirable to reduce the value of the α function when manyflows are traversing the same bottleneck buffer, and raise it again ifthe number of flows decreases. Unfortunately, without collaboration fromthe bottleneck, it is not possible to determine the number of flows.Furthermore, in general, not all the flows might be willing to send atthe same rate, so a simple count of the flows would not be the bestinput to the α function.

Consequently, the techniques and mechanisms of the present disclosureuse the forward delay component to adapt the α function. According tovarious embodiments, a simple proportional controller is used, althoughmore complex controllers such as proportional integral derivative (PID)controllers can also be used. As noted above in Equation 3, givena-priori knowledge of the bottleneck bandwidth, a target bufferoccupation L is compared to an estimate of the current buffer occupationobtained from the measured qdelay. An increase in qdelay is considered aconsequence of increased amounts of traffic, generated by other sources,converging on the bottleneck and casing the bottleneck buffer occupationto grow. The proportional controller in Equation 3 applies negativefeedback to the system, reducing α and consequently reducing thecontribution of each flow to the bottleneck buffer utilization. Sincethe measured quantity to calculate α is the qdelay, a quantity whoseaverage is going to be the same for all the connections sharing a singlebottleneck, it is guaranteed that all these connections, on equilibrium,will tend to use the same value of α, resulting in good fairnessproperties.

An Additive Increase/Multiplicative Decrease (AIMD) scheme similar tothe one used in conventional TCP was evaluated in order to adjust α, butit was difficult to obtain acceptable fairness. To achieve it inreasonable timescales, the rate of adjustment needs to be high, but ahigh rate of adjustment causes oscillations in the bottleneck queue sizethat can lead to packet drops.

Consequently the techniques of the present disclosure recognize that aproportional controller preferably has both good fairness and a quickresponse to changing network conditions. According to variousembodiments the techniques of the present disclosure provide aproportional controller that allows a system to reach a stableequilibrium point in which all the connections on a bottleneck achievethe same constant throughput, not just a variable throughput whoseaverage is the ideal one. The proportional controller also calculates anew value for every RTT and is based on the last estimate of the queuingdelay. This is important to avoid drops when network conditions change,for example when a new flow activates, and to be able to consume all theavailable bandwidth when flows depart. In typical networks, there areusually other sources of delay besides congestion and propagation times,and in order to overcome these extra delays a minimum positive α is usedto achieve full rate.

While FastTCP calculates a new congestion window every RTT, thetechniques of the present disclosure recognize that it is notnecessarily a good idea to immediately increase the window to the newvalue, because the resulting burst could cause packet drops and resultin extra queuing delay that increases the measured RTT. This would occureven though the window is still smaller than the one that would bereached on equilibrium. Furthermore, bursts cause decreased performancesince the increased RTT, even though temporary, limits the growth of thewindow towards the value it would reach at equilibrium.

Consequently, the techniques of the present disclosure reduce burstinessby applying variable rate shaping and by allowing the congestion windowto grow only gradually. The packet transmission process is shapedaccording to the following equation:max_rate=(snd_hiwat/base_rtt)(1+1/32)  (5)where max_rate is a transmission rate, snd_hiwat is the maximum window;and base_rtt is the observed minimum round trip time. Shaping isimplemented by smoothly delaying packet departures in order to obtainthe desired send rate. Of course new packets are sent only if theybelong to the usable portion of the window.

According to various embodiments, techniques of the present disclosurenot only provide more efficient and effective ways for determiningcongestion window sizes but also provide a variable shaper to smoothbursts of data. The shaper is programmed to a rate 1/32 faster than theaverage one achievable using the new target window, but still shapes ata rate close enough to the achievable rate in order to better spread thetransmission of packets. The correcting factor allows full utilizationof the usable window portion. According to various embodiments, shapingis important when the target window is lower thanbottleneck_bandwidthrtt, i.e., below equilibrium, or in presence ofother competing TCP connections. It should be noted that even if thecongestion window is held constant, and the packets are initially evenlyspread across the whole RTT, packets tend to cluster in time in theabsence of shaping, resulting in rate variations from the average, andnetwork buffer oscillations.

According to various embodiments, the congestion window is updated forevery positive acknowledgment received. The change in the congestionwindow is calculated using the following equation:

$\begin{matrix}{\mspace{79mu}{{{\Delta\;{cwnd}} \cong {\left\lbrack {\min\left( {{{snd\_ hiwat} - {cwnd}},\alpha} \right)} \right\rbrack/4}}\left\{ \begin{matrix}\begin{matrix}{\left. {{L - {{max\_ bw}*{qdelay}}} > 0}\Rightarrow\alpha \right. = {\left( {L - {{max\_ bw}*{qdelay}}} \right) \cdot K_{p}}} & {0 < K_{p} \leq 1}\end{matrix} \\{\left. {{L - {{max\_ bw}*{qdelay}}} \leq 0}\Rightarrow\alpha \right. = 0}\end{matrix} \right.}} & (6)\end{matrix}$where cwnd is a congestion window; snd_hiwat is the maximum window;qdelay(t) is the forward delay component of the round trip time; L is atarget buffer occupation; max_bw is a maximum bandwidth and K_(p) is anempirically determined proportional controller constant between 0 and 1.

However, the techniques and mechanisms of the present disclosurerecognize that growing the congestion window using conventional TCPmechanisms is inadequate. Growth according to available TCP mechanismsis too slow and reduction is too drastic. Consequently, the techniquesof the present disclosure provide for more gradual changes to congestionwindow sizes.

The increase in congestion window size occurring every RTT is reduced inorder to dampen the window oscillation dynamics by a factor of ¼. Itshould be noted that a variety of reduction factors can be used, but 4is a hardware bit shifting friendly reduction factor. Besides thereduction to ¼ of the gap between snd_hiwat and cwnd, this relation alsocovers scenarios after an idle period where the gap between snd_hiwatand cwnd is quite large. After an idle or partially idle period (lastingless than one rtt), snd_hiwat reflects a window used in the past andcwnd tracks the amount of data that is still pending in the network, inorder to reduce burstiness. In this case, it is not desirable to growcwnd towards snd_hiwat, since snd_hiwat does not appropriately representthe network congestion state. In such cases the cwnd is only grown byα/4.

Another corner case can happen after an IPS FastTCP failure, i.e., aftera drop event and the subsequent recovery. In such a case, it is commonthat a large number of bytes is acknowledged by a single packet, so thenumber of bytes acknowledged could have a value significantly largerthan normal, resulting in a sudden window growth that could cause otherdrops. To improve the stability of IPS FastTCP after a drop event, thenumber of bytes acknowledged is limited. According to variousembodiments, the variable holding the number of bytes acknowledged, whenused to grow the congestion window cwnd, is limited to 16000 bytes. Thisvalue has been empirically chosen, but different values might proveeffective. Of course the actual value of acked is used as usual to slidethe window and discard newly acknowledged data.

The above Equation 6 gives only the target cwnd growth over the nextround trip time. To perform a gradual increase of cwnd, the followingequations can be used:

$\begin{matrix}{\mspace{79mu}{{{\Delta\;{actcwnd}} = \frac{\frac{1}{4}{\min\left( {{{snd\_ hiwat} - {cwnd}},\alpha} \right)}}{cwnd}}\left\{ \begin{matrix}\begin{matrix}{\left. {{L - {{max\_ bw}*{qdelay}}} > 0}\Rightarrow\alpha \right. = {\left( {L - {{max\_ bw}*{qdelay}}} \right) \cdot K_{p}}} & {0 < K_{p} \leq 1}\end{matrix} \\{\left. {{L - {{max\_ bw}*{qdelay}}} \leq 0}\Rightarrow\alpha \right. = 0}\end{matrix} \right.}} & (7) \\{\mspace{79mu}{{cwnd} = {{cwnd} + {\Delta\;{{actcwnd} \cdot {acked}}}}}} & (8)\end{matrix}$

Equation 8 allows a gradual update every time a positive acknowledge isprocessed. However, more state information than usual needs to be keptin the TCP control block.

The techniques of the present disclosure also recognize that in someinstances, the baseRTT variable may have to be recalculated. Forexample, the base round trip time for a particular source destinationpair may change when network topology changes. One trivial case happenswhen the delay is decreasing. In that case it is sufficient to use thenew observed RTT as the new baseRTT. A difficult case is one thatresults from increases in the propagation delay, because the increasecould also result from network congestion. The techniques of the presentdisclosure recognize that the variable baseRTT should not be increasedas a result of network congestion.

According to various embodiments, the TCP stack keeps track of tworecently observed RTTs in order to properly update the baseRTT variableand uses the following equation:baseRTT=min(min_rtt,prev_min_rtt)  (9)where prev_min_rtt is the best estimate of baseRTT on a period prior tothe previous 30 seconds epoch; and min_rtt is the smallest RTT observedin the last 30 seconds epoch.

According to various embodiments, the prev_min_rtt and min_rtt arecompared periodically. In some examples, the values are compared every30 seconds. Only if min_rtt is sufficiently bigger than prev_min_rtt,the value of prev_min_rtt is min_rtt, otherwise no change is made. Therationale is that a large amount of congestion is unlikely to persistfor 30 seconds, so during the 30 second interval the measured round triptime should oscillate and reach a value sufficiently close to thepreviously observed minimum. If instead the increased round trip time isdue to a longer path, the observed RTT does not oscillate close to theold minimum, but is more likely to be exactly the new (bigger) minimum,since congestion is unlikely because all connections are trying to adaptto the longer path. In the presence of pre-existing congestion on thenew path, the observed RTT will still oscillate, but will be bigger thanthe one measured on the old path. If the baseRTT is overestimated, dropsare likely, but the subsequent path underutilization will result in agood baseRTT estimate.

In some examples, the min_rtt value can be updated using the followingequations:min_rtt−prev_min_rtt>min_delta

prev_min_rtt=min_rtt  (10)min_delta=L*4/max_bw  (11)where min_delta is the minimum time increase that is considered stilldue to congestion.

The min_rtt value can be adjusted when significant routing changesoccur.

It should be noted that the techniques of the present disclosure canalso be implemented by traffic destinations. In some examples, a sourcecan control the send process of a destination by modulating windowsassociated with the destination. The techniques of the presentdisclosure can be used to pace acknowledgements to a destination so thatwindows at a destination can be modulated based on computations at asource.

The techniques of the present disclosure can be implemented on a varietyof devices such as hosts and switches. In some examples, theimprovements to TCP can be implemented at any source originating trafficor destination receiving traffic. To be effective, it does not need tobe implemented on both. In other examples, the improvements to TCP canalso be implemented at tunneling switches used to transmit storageapplication data over IP networks.

FIG. 5 is a diagrammatic representation of one example of a fibrechannel switch that can be used to implement techniques of the presentdisclosure. Although one particular configuration will be described, itshould be noted that a wide variety of switch and router configurationsare available. The tunneling switch 501 may include one or moresupervisors 511. According to various embodiments, the supervisor 511has its own processor, memory and storage resources.

Line cards 503, 505, and 507 can communicate with an active supervisor511 through interface circuitry 583, 585, and 587 and the backplane 515.According to various embodiments, each line card includes a plurality ofports that can act as either input ports or output ports forcommunication with external fibre channel network entities 551 and 553.The backplane 515 can provide a communications channel for all trafficbetween line cards and supervisors. Individual line cards 503 and 507can also be coupled to external fibre channel network entities 551 and553 through fibre channel ports 543 and 547.

External fibre channel network entities 551 and 553 can be nodes such asother fibre channel switches, disks, RAIDS, tape libraries or servers.It should be noted that the switch can support any number of line cardsand supervisors. In the embodiment shown, only a single supervisor isconnected to the backplane 515 and the single supervisor communicateswith many different line cards. The active supervisor 511 may beconfigured or designed to run a plurality of applications such asrouting, domain manager, system manager, and utility applications.

According to various embodiments, the switch also includes line cards575 and 577 with IP interfaces 565 and 567. In one example, the IP port565 is coupled to an external IP network entity 555. According tovarious embodiments, each IP line card includes a plurality of portsthat can act as either input ports or output ports for communicationwith external IP entities 555. These IP entities could be IP switches orrouters, or directly attached network endnodes. The line cards 575 and577 can also be coupled to the backplane 515 through interface circuitry595 and 597.

According to various embodiments, the switch can have a single IP portand a single fibre channel port. In one embodiment, two fibre channelswitches used to form an FCIP tunnel each have one fibre channel linecard and one IP line card. Each fibre channel line card connects to anexternal fibre channel network entity and each IP line card connects toa shared IP network.

In another embodiment, a fibre channel switch performs gateway functionsbetween hosts or disks connected to an IP network and host or disksconnected to a fibre channel network. In various embodiments, thetechniques of the present disclosure do not need to be implemented byboth TCP end points in order to be effective.

In addition, although an exemplary switch is described, theabove-described embodiments may be implemented in a variety of networkdevices (e.g., servers) as well as in a variety of mediums. Forinstance, instructions and data for implementing the above-describeddisclosure may be stored on a disk drive, a hard drive, a floppy disk, aserver computer or a remotely networked computer. Accordingly, thepresent embodiments are to be considered as illustrative and notrestrictive, and the disclosure is not to be limited to the detailsgiven herein, but may be modified within the scope and equivalents ofthe appended claims.

While the disclosure has been particularly shown and described withreference to specific embodiments thereof, it will be understood bythose skilled in the art that changes in the form and details of thedisclosed embodiments may be made without departing from the spirit orscope of the disclosure. For example, embodiments of the presentdisclosure may be employed with a variety of network protocols andarchitectures. It is therefore intended that the disclosure beinterpreted to include all variations and equivalents that fall withinthe true spirit and scope of the present disclosure.

The invention claimed is:
 1. A method for performing congestion controlusing a transmission control protocol (TCP), the method comprising:determining a forward delay component of a round trip time associatedwith sending data associated with a flow from a source node andreceiving an acknowledgment from a destination node using thetransmission control protocol (TCP); determining a maximum window basedon at least: (i) the forward delay component; (ii) a minimum round triptime observed between the source node and the destination node; (iii) aprevious congestion window for sending the data from the source node tothe destination node; (iv) a maximum bandwidth between the source nodeand the destination node; and (v) a target buffer amount for the flow toconsume; and transmitting data using the maximum window.
 2. The methodof claim 1, wherein the maximum bandwidth is the capacity of the slowestlink along the path from the source node to the destination node.
 3. Themethod of claim 1, wherein determining the forward delay componentcomprises determining the reverse delay component.
 4. The method ofclaim 1, wherein the previous congestion window is updated using anamount of data acknowledged.
 5. The method of claim 4, wherein thechange in the congestion window is determined using the maximum window,the previous congestion window, the forward delay component, and amaximum bandwidth.
 6. The method of claim 1, wherein a variable rateshaper is used to determine a transmission rate.
 7. The method of claim6, wherein the transmission rate is determined using the maximum windowand the observed minimum round trip time.
 8. A network device forperforming congestion control using a transmission control protocol(TCP), the network device comprising: an interface coupled to anInternet Protocol (IP) network; and a processor for determining aforward delay component of a round trip time associated with sendingdata associated with a flow from a source node and receiving anacknowledgment from a destination node connected to the interface usingthe transmission control protocol (TCP), wherein the processordetermines a maximum window based on at least: (i) the forward delaycomponent; (ii) a minimum round trip time observed between the sourcenode and the destination node; (iii) a previous congestion window forsending the data from the source node to the destination node; (iv) amaximum bandwidth between the source node and the destination node; and(v) a target buffer amount for the flow to consume; and wherein theprocessor transmits data using the maximum window.
 9. The network deviceof claim 8, wherein the maximum bandwidth is the capacity of the slowestlink along the path from the source node to the destination node. 10.The network device of claim 8, wherein determining the forward delaycomponent comprises determining the reverse delay component.
 11. Thenetwork device of claim 8, wherein the previous congestion window isupdated using the amount of data acknowledged.
 12. The network device ofclaim 11, wherein the change in the congestion window is determinedusing the maximum window, the previous congestion window, the forwarddelay component, and the maximum bandwidth.
 13. The network device ofclaim 8, wherein a variable rate shaper is used to determine atransmission rate.
 14. The network device of claim 13, wherein thetransmission rate is determined using the maximum window and theobserved minimum round trip time.
 15. A system for performing congestioncontrol, the system comprising: means for determining a forward delaycomponent of a round trip time associated with sending data associatedwith a flow from a source node and receiving an acknowledgment from adestination node using the transmission control protocol (TCP); meansfor determining a maximum window based on at least: (i) the forwarddelay component; (ii) a minimum round trip time observed between thesource node and the destination node; (iii) a previous congestion windowfor sending the data from the source node to the destination node; (iv)a maximum bandwidth between the source node and the destination node;and (v) a target buffer amount for the flow to consume; and means fortransmitting the data using the maximum window.
 16. The system of claim15, wherein determining the forward delay component comprisesdetermining the reverse delay component.
 17. The system of claim 15,wherein the previous congestion window is updated using the amount ofdata acknowledged.
 18. The system of claim 17, wherein the change in thecongestion window is determined using the maximum window, the previouscongestion window, the forward delay component, and the maximumbandwidth.
 19. The system of claim 15, wherein a variable rate shaper isused to determine a transmission rate.
 20. The system of claim 19,wherein the transmission rate is determined using the maximum window andthe observed minimum round trip time.